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Introduction 


The purpose of this publication is to assist State and local education agencies in defining the evaluation 
process for a Request for Proposal (RFP). It offers a suggested structure to the evaluation that allows 
input from a wide range of participants and specific subject matter experts, permits flexibility 

and weighted scoring in appropriate areas and provides an objective and defensible process for 
determining the vendor finalist. 


The first step is to determine the major components of the RFP to be scored and publish that high-level 
evaluation method in the RFP document itself. A sample evaluation breakdown might be as follows: 


Written Proposal 15% 
Functional Requirements 25% 
Cost 20% 
Vendor Demonstrations 35% 
Reference Check 5% 


The vendor demonstrations comprise the largest weight in the scoring process. This is intentional 
because not only can the evaluation team see the product first hand and make its own judgments, 

but also the team can get an understanding of the proposing vendor, their agency and the proposed 
implementation team. The percentages above can vary based on the preference of the agency, but it is 
recommended that the demonstrations be the largest component of the evaluation. 


A wide range of stakeholders should contribute to the evaluation of the vendors and their proposed 
solutions. Not all participants need to be involved in every component but it is wise, depending on 
the size of the agency, to have a core team of 8-10 individuals. Table 1 provides an example of the 
subgroups and their participation level in the RFP evaluation and vendor selection process. 


Table 1 
Stakeholders 
Group Makeup Involvement 
Core Evaluation Team Group of 8 — 10 individuals who = Read and score written proposals 
eee evekyatieatet = Review functional requirement responses 
= Review cost response 
= Evaluate vendor finalists’ demonstrations 
= Participate in final vendor selection 
Demonstration Evaluation Committee Group of 20 — 30 people (including core = Evaluate vendor finalists’ demonstrations 
eyallietion teat} = Participate in final vendor recommendation 
Agency's Leadership Cabinet or director-level leadership = Act on recommendation of Evaluation Committee 


The scoring methodology below this high level would not be provided in the RFP document but 
should be clearly thought out and decided upon in advance. Vendor responses should be evaluated 
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relative to each other rather than against some exact set of criteria. One method to accomplish this is 
to assign point values to the various components listed above, standardize to some equal number the 
maximum value for each component, assign the highest scoring vendor the highest point value for the 
section, prorate the remaining vendor scores based on the best response, and then multiply the point 
values by the percentage associated with each component. 


Because the vendor demonstrations require a major commitment of time and personnel for both the 
requesting agency and the vendors, a two-phase process for the evaluations is recommended. Phase 1 would 
consist of the first three components listed above. Once the agency has evaluated the written proposals, 
the vendor self-reported ability to meet the functional requirements, and the cost of the solution, two or 
three vendor finalists can be chosen to move to the next phase of on-site demonstrations and in-depth 
reference checks. 


Before detailing the scoring methods and tools for each section, the overall, end-result scoring might look 
like what is shown in Tables 2-4. 


Table 2 
Phase 1 Scoring Matrix 


A ; , ; 

5-year Cost of Be Normalized Appendts Nopusied Miter Normalized 

Vendor : Cost ; 1Raw ~~ Appendix 1 RFP Raw 
Ownership ; Cost Points Raw Score 
Points Score Score Score 

Vendor 1 N/A 0.00% 0 318 550 23 | 
Vendor 2 $1,038,600 92.08% 921 184 479 829 207 242 767 ees) 506 
Vendor 3 $987,593 96.83% 968 194 577 1000 250 315 1000 150 594 
Vendor 4 $5,703,825 16.77% 168 34 485 841 210 283 898 135 379 
Vendor 5 $1,129,789 84.65% 846 169 559 968 242 303 960 144 555 
Vendor 6 $956,335 100.00% 1000 = 200—Sts«iS21 903 226 249 788 ie | 544 
Table 3 
Phase 2 Scoring Matrix 


Phase 2 


Reference Normalized 


Vendor ie ge se Check Reference 
Score Check Score 

Vendor 1 
Vendor 2 
Vendor 3 578 767 268 1 95 969 7 48 | 911 
Vendor 4 
Vendor 5 754 1000 350 98 1000 50 955 ies 
Vendor 6 455 603 f P 211 ‘| 93 949 47 i 803 Bi 
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Table 4 
Total RFP Evaluation Scoring Matrix 


Phase 1 


5-year Cost of ey Normalized ppvetale Honmale = aie! Normalized 
Vendor ; Cost : 1Raw Appendix 1 RFP Raw 
Ownership ; Cost Points Raw Score 
Points Score Score Score 


Vendor 1 N/A 0.00% 

Vendor 2 $1,038,600 92.08% 921 184 479 829 207 242 767 115 506 
Vendor 3 $987,593 96.83% 968 194 577 1000 250 315 1000 150 594 
Vendor 4 $5,703,825 16.77% 168 34 485 841 210 283 898 135 379 
Vendor 5 $1,129,789 84.65% 846 169 559 968 242 303 960 144 555 
Vendor 6 $956,335 100.00% 1000 200 521 903 226 249 788 118 544 


: Reference Normalized 
Demo Raw Normalized 
Vendor Check Reference 
Score Demo Score 
Score Check Score 
Vendor 1 
Vendor 2 
Vendor 3 578 767 268 95 969 48 911 ed 
Vendor 4 
Vendor 5 754 1000 350 98 1000 50 955 | is 


Vendor 6 455 603 m 93 949 47 803 Pl 


In the example, all components are normalized to 1,000 points and each vendor's score is computed as 
described above. The major component scores are highlighted in yellow. In this case, the top three vendors, 
based on the cost, response to functional requirements, and written proposal, were selected to move on 

to the second phase. The scores for the vendor demonstrations and reference checks were added to the 
previous scores to get a total point summary and vendor rank. 


Following is a more detailed explanation of each major component and the tools and methods to compute 
the score for each vendor. 


Phase 1 
Written Proposal Evaluation 


The process to review, analyze and evaluate a written proposal is an important and time-consuming task. 
The RFP itself should have a clearly defined section that outlines what the agency wants the vendors to 
provide in their proposals and be specific on the order in which their responses should be structured. This 
orderly and controlled structure will aid the evaluation team as it reviews the materials from each proposal. 


An evaluator’s guide should be developed to mirror the structure of what is being asked for in the written 
proposals and include guiding questions. Each major section can be organized into smaller components and 
weighted as desired. It is not required that all members of the evaluation team read every section of the proposal. 
For example, if there are members of the core evaluation team who are not comfortable with the technical 
aspects of the RFP, they can skip the technical sections and allow those more qualified to evaluate those sections. 


Table 5 gives a sample of part of a guide that shows a section with questions that match what is asked in the RFP. 
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Table 5 
Evaluator's Guide 


Question Response 
aapre Addressed Quality Comments Score 


0-10 
v L-M-H 


Narrative Response to Functional Requirements (Section 3.2) (50 points) 
Standards and Curriculum — 10 points 


+ Does the proposal provide information on how it will support multiple sets and structures of 
standards such as the CCSS and NJCCCS? 


+ Does the proposal describe how it will support the NJ Model Curriculum? 


+ Are workflow processes for submittal and approval of curriculum materials based on roles 
and permissions included? 


+ Does the proposal describe how additional educational and professional development 
teaching resources outside the system may be managed and aligned to standards and SLOs? 


+ Does the proposal provide a best practice model curriculum environment that is well vetted 
and mapped to the existing CCSS and NJCCCS? 


Instructional Design and Practice — 10 points 
+ Does the proposal include a module for lesson plan development, storage, and delivery? 
+ Is the ability to provide differentiated instruction described? 


+ Does the proposal include the ability for teachers to group students by type of instruction 
required, district, school, classroom, teacher, grade, standard, and SLO? 


+ Does the proposal support the assignment of metadata tags to educational materials to 
allow for standardized searches? 


A guide allows evaluators to take notes as they read each section of the RFP responses. A rubric is designed 
to be used to enter a score, in this example 0-10, for each of the subsections of the response. Evaluators 
may want to read all of the vendor responses before going back and entering scores based on the quality of 
the response. 


A custom rubric for every section and question in the guide is not required because a generic rubric can be 
created and used for all sections. Even though a score of 0-10 is allowed in the example evaluator’s guide, 
the score may be based on a number of generic factors such as: 


* Have all the questions been answered completely? 


- Is the quality of the response high (depending on reviewer's expectation)? 


- Did the proposer provide additional detail to further clarify the answer? 


+ Did the proposer demonstrate a thorough understanding of the questions? 


Reviewers might also want to include notes for each section to help them justify the score they give for the 
section. Some procurement policies require all evaluation tools and templates, including notes, be publicly 
documented and available for review following the vendor selection. Reviewers should use care in any 
remarks collected on this evaluation form. 


The scores from all evaluators can then be put into a written evaluation summary to determine the point 
total for each vendor for the written response component. All evaluators’ scores for each component can be 
averaged and multiplied by the topic weight to give the total points for each subsection. Table 6 shows an 
example of a section of the written RFP evaluation summary. 
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Table 6 
Written RFP Evaluation Summary 


Vendor Vendor 1 Vendor 2 Vendor 3 


Written RFP - 300 points total Name Name ETT 


Narrative Response to Functional Requirements - 50 points 


Standards and Curriculum 10 9 8 
Instructional Design and Practice 9 9 8 
Assessment and Growth 3 8 10 
Data Analysis and Reporting 8 10 6 
User Interface 8 9 2 
Total 38 45 34 


Narrative Response to Technical Requirements - 70 points 


Physical Structure and Location 5 4.5 4 
Data Ownership 4 3 3 
Technical Contractor Partnering 5 5 4 
Security Strategy 5 2 4 
Backup and Recovery 3 2 3 
Single Sign-on 5 2 4 
Client Architecture 5 45 4 
Availability and Reliability 9 7 8 
Accessibility and Usability 4 3 2 
Proposed Approach to Enhancements and Customizations 4 4 4 
Transition and Integration Strategy 5 5 2 
Software Support and Maintenance 5 4 4 
Customer Call Center (Help Desk) Management 5 4 4 
Total 64 50 50 


Each vendor's points would be totaled and entered into the appropriate column on the RFP evaluation 
scoring spreadsheet, as shown in Table 2. 
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Functional Requirements Evaluation 


The second component of the first phase of evaluation is the vendor's responses to the functional 
requirements of the RFP. Normally, there are hundreds of these requirements subdivided into the major 
areas that the proposed solution is to address. Because there are too many of these discrete requirements 
to evaluate individually, this portion of the proposal becomes more of a self-reporting evaluation by the 
vendor of their products’ ability to meet each requirement. 


To foster accurate reporting by the vendors, two things should be made clear: each requirement response will 
become part of the contract of the successful vendor and the vendor must stipulate the extent to which it meets 
the requirement. The “extent” responses should be clearly identified in the RFP documents as to what responses 
are valid. For example, a set of valid responses to each functional requirement might be as shown in Table 7. 


Table 7 
Functional Requirement Response 


Response | Definition 


Y—Yes Requirement will be met. This application requirement is met by proposed software that is installed and operational at other sites and can be 
demonstrated to the district. 


N—No Requirement will not be provided. 
Requirement will be met by customizing existing software or through the use of software tools 
such as application report writer or query, at no cost to the State education agency or local education agency. 


Note: In the Comments column next to this response, you must indicate the following: 
C1—Customization 

+ Description of customization 

+ Estimated level of complexity (High, Medium, Low) 

+ Target date for completion 


Requirement will be met by customizing existing software for an additional cost to the district. 


Note: In the Comments column next to this response, you must indicate the following: 
+ Description of customization 
(2—Customization P 
+ Estimated level of complexity (High, Medium, Low) 
+ Target date for completion 
+ Estimated associated costs (must also be included in the RFP Costs Forms) 


Requirement will be met by packaged software that is currently under development, in beta test or not yet released. Please indicate target date for 


UD—Under Development : 
completion. 

Requirement will be met by third-party software package and is included at no additional cost in this proposal. 

coe Note: In the Comments column, indicate the name of the proposed third-party software package and indicate the interface/integration services 


being proposed. 


A value can be assigned to each of the response codes to indicate the percentage of credit to be given for each 
requirement. For example, Table 8 shows what the weight for the vendor's responses in the example might be: 


Table 8 
Response Code Values 


Code Weight 


Y 1.0 
N 0.0 
q 0.8 
Q 0.2 
UD 0.6 
3 1.0 
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This percentage can then be multiplied by the weight of each requirement determined by the agency. An 
example of such a result is shown in Table 9 


Table 9 
Sample Functional Requirements Evaluation 


Section 1 Weighted 


Score 


System utilizes multiple sets of standards with differing structures, 
hierarchy, and terminology (e.g. Common Core, career/vocational tech, 4.0 Yes 4 Q 0.8 UD 2.4 
state, district, content standards, ELP standards, etc.) 


Allows for variability in structures among different sets of standards (e.g. 
content area, level, strand, standard, topic, SLO, benchmark) 


Provides for the definition and customization of standards terminology (e.g. % 
3.0 0 3 3 Yes 3 
redefine strand to another term) 


User with the appropriate rights will have the ability to import fully 


developed database of existing standards af ve Ne 4 ue 2 
Ability to manage multiple levels of content standards (i.e. content area, 
level, strand, standard, topic, SLO, benchmark, cumulative progress 4.0 Yes 4 Yes 4 Yes 4 


indicator) 


While the color coding of the example spreadsheet can indicate at a glance the function most covered by 
the vendor's product, each major section of the functional requirements may also have a relative weight 
in determining the total points for this component. The total score for each of the functional requirements 
then is also entered into the Appendix 1 Raw Score column on the RFP evaluation scoring spreadsheet, as 
shown in Table 2. 


Cost 


One of the hardest areas to structure in order to compare “apples to apples’ is the cost section. Each 
vendor may have a different approach, pricing scheme or volume discount incentives. Models for 
solutions deployment continue to evolve to include hosted solutions and software-as-a-service beyond 
the traditional, agency-hosted solutions. The cost forms must clearly identify any one-time license fees 
or ongoing maintenance costs. If possible, the forms should be designed in a modular fashion by asking 
the vendor to provide discrete pricing for each module of the proposal. This will allow unbundling of 
the proposed solution if it is in the best interest of the State education agency (SEA) or local education 
agency (LEA). 


Set up, implementation, training, support/maintenance and upgrades should all be components of the 
cost proposal. 


A single calculated cost for each vendor will be required to compare that vendor to the others. It is 
suggested that the agency compare 5- or 10-year cost of ownership for this purpose. Once that cost 

is determined for each vendor, the same algorithm that is used to give prorated cost points should be 
followed and the figures entered into the appropriate column on the RFP evaluation scoring spreadsheet, as 
shown previously in Table 2. 
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Vendor Finalists Selection 


Once the three components of Phase 1 have been evaluated, the numbers are entered into the summary 
spreadsheet and the top two or three vendors can be selected to move on to Phase 2. At this point, the 
summary sheet in Table 2 would appear as shown in Table 10. 


Table 10 
Phase 1 Evaluation Results 


% of Appendix —_ Normalized Written 


wt get Mem let ye ee al 
Points Score Score Score 
Vendor 1 N/A 0.00% 0 318 550 23 a 
Vendor2 $1,038,600 92.08% 921 184 479 829 207 242 767 115 506 
Vendor 3 $987,593 96.83% 968 194 577 1000 250 315 1000 a ll 
Vendor4 $5,703,825 ‘16.77% 168 34 485 841 210 283 898 ao 7 am 
Vendors $1,129,789 84.65% 846 169 559 968 242 303 960 mw ss i 
Vendor 6 $956,335 100.0% 1000 200 521 903 226 249 788 7 o« 


Phase 2 
Vendor Demonstrations 


Vendor demonstrations are the most important aspect of the evaluation process and give the SEA or LEA an 
opportunity to see how the vendor solution meets the critical components of the RFP. To accomplish this, a 
demonstration script should be developed with input from the subject matter experts in each component 
of the RFP. Vendors should be asked as much as possible to demonstrate, rather than describe, how the 
solution would be used in the agency to accomplish the desired objectives. The development of the script 
eliminates the “dog and pony show" that some vendors may want to provide. This ensures that they will 


” 


address the real needs of the agency rather than focusing on “bells and whistles. 


The agenda should be created by working with all stakeholders to determine the amount of time required 
to adequately evaluate the concerns of each component. It is not necessary that all evaluators attend all 
components of the demonstrations but, if they are to evaluate a section of the demonstration, they must 
attend and evaluate that section for all vendors. The agenda should be structured to be as unobtrusive as 
possible into the normal duties of the evaluators. For example, the SEA and/or LEA business and technical 
components could be addressed on one day and the second day devoted to the use of the system by 
principals, teachers, parents and students. 


Reference Checks 


The second component of Phase 2 of the evaluation is reference checking. Typically, vendors are asked in 
the RFP to provide a list of 3-5 references of size, complexity, and purpose similar to those of the requesting 
SEA or LEA. One strategy for reference checks is to send a short questionnaire to be completed by the 
contact person at each reference site. That person would answer the questions and email them back to the 
team at the agency, who would then schedule a follow-up teleconference. 


For the questionnaire, two or three questions should be developed for a number of implementation areas. 
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These areas could include: 

+ Background (scope of work, modules purchased, length of use) 
Quality of Planning (agency and vendor planning process) 
Stakeholder Communications (process, vendor participation, effectiveness) 
Execution of Overall Plan (vendor execution, issue resolution, agency advice) 
Training Plan (type of training, extent of training, agency recommendations) 
Project Benchmarks and Timeline (adherence, lessons learned) 
Adherence to Budget (any cost overruns, required agency personnel, total cost of ownership) 
Customer Support (vendor responsiveness, type of support, hours of operation) 
System Performance (performance issues, software deficiencies, concerns) 
Overall Satisfaction (agency level, school level, individual level) 


To provide both quantitative and qualitative data from the reference checks, each of the implementation 
areas could be given a score. One suggested rubric would be: 


5 - Excellent 
4-Very Good 
3 — Average 

2 — Fair 

1 — Poor 


The quantitative data can then be entered into a spreadsheet and the results displayed in multiple formats. 
Two formats are shown in Table 11 and Figure 1. 


Table 11 
Vendor Reference Check Results 


Quality of Planning 4.92 4.67 4.67 
Stakeholder Communications 5.00 4.33 4.75 
Execution of Overall Plan 4.65 4.50 4.50 
Training Plan 4.77 3.67 4.17 
Project Benchmarks and Timelines 4.17 4.50 5.00 
Adherence to Budget 4.83 5.00 5.00 
Customer Support 4.58 5.00 4.75 
System Performance 4.83 4.75 4.33 
Overall Satisfaction 4.75 4.67 4.50 
Total Average: 4.72 4.56 4.63 
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Figure 1 
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Vendor Finalist Selection 


As described in each section of Phase 1, each vendor's points for the two Phase 2 sections would be totaled 
and entered into the appropriate column on the RFP evaluation scoring spreadsheet, as shown previously in 
Table 2 and in Table 12. 


Table 12 
Evaluation Scoring for Phases 1 and 2 


Phase 1 


% of -20%of — Appendix Normalized  25%of Written 


Vendor suai Cost boli Cost 1Raw Appendix 1 A dix RFP Raw ballin 
Points Points Score Score nt Score 

Vendor 1 N/A 0.00% 0 0 318 550 138 23 74 11 149 | 6 
Vendor 2 $1,038,600 92.08% 921 184 479 829 207 242 767 115 506 4 
Vendor 3 $987,593 96.83% 968 194 577 1000 250 315 1000 150 594 1 
Vendor 4 $5,703,825 16.77% 168 34 485 841 210 283 898 135 379 5 
Vendor 5 $1,129,789 84.65% 846 169 559 968 242 303 960 144 555 2 
Vendor 6 $956,335 100.00% 1000 200 521 903 226 249 788 118 544 3 


Phase 2 


Vendor pee se ee : ce le ee cn 
atten Score Check Score 
Vendor 1 
Vendor 2 
Vendor 3 578 767 268 95 969 48 911 2 
Vendor 4 
Vendor 5 754 1000 350 98 1000 50 955 1 
Vendor 6 455 603 211 93 949 47 803 3 
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These results clearly indicate that Vendor 5 is the leading candidate for recommendation by the evaluation 
committee to the agency leadership. 


As a final validation and documentation, it is suggested that the members of the evaluation team sign off on 
the recommendation. A sample of a signoff form is provided in Table 13. 


Table 13 
Committee Signoff Sheet 


As amember of the RFP proposal review team and/or the vendor demonstration evaluation team, | support the 
decision of the [application] selection committee to award the bid to the vendor circled below: 


Vendor 1 Vendor 2 Vendor 3 


Name Team (check all that apply) Department/Position 


RFP review team 
Demo evaluation team 


RFP review team 
Demo evaluation team 


RFP review team 
Demo evaluation team 


RFP review team 
Demo evaluation team 


RFP review team 
Demo evaluation team 


RFP review team 
Demo evaluation team 


Evaluation Timeline 


The RFP evaluation process typically takes 10—12 weeks depending on the complexity of the application, the 
number of participants and the procurement policies and practices of the agency. An estimated schedule 
from vendor submission of proposals to contract signing is typically included in the RFP. The vendor 
negotiations stage is difficult to estimate and may take much longer than anticipated. A sample timeline and 
milestones based on the due date of the proposals are outlined in Table 14. 


Table 14 
Evaluation Timeline 


Milestone | Date 


Proposals submitted to organization Proposal Due Date 
Evaluation of written proposals +2 weeks 
Selection of vendor finalist(s) +1 week 

(Vendor finalists prepare for demonstrations) Vendor reference checks by agency +2 weeks 

Vendor finalist(s) demonstrations +2 weeks 

Vendor selection +1 week 

Vendor contract negotiations +3-4 weeks 
Work begins +2 weeks 
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In the example outlined in Table 14, Phase 1 of the evaluation process takes about three weeks from 
proposal submission. Once two to three vendors are chosen as finalists to move to Phase 2, the agency 
can begin conducting the reference checks during the same time that the vendors are preparing for their 
demonstrations. Therefore, the demonstration script must be ready to be distributed to the vendors soon 
after the submission date. Vendor demonstrations normally can take place within a two-week window 
depending on the number of vendors and the length of the demonstrations. An additional week is built 
into the schedule to allow the evaluation team to consolidate all of the information from the components 
of both phases and come to a consensus on the vendor they choose to recommend to the agency's 
leadership. Three to four weeks are then reserved for contract negotiations. Finally, although a work start 
date is anticipated by both the vendor and agency, two weeks are normally needed to schedule and 
prepare for the project kickoff. 


This publication features information from public and private organizations. Inclusion 
of this information does not constitute an endorsement by the U.S. Department 
of Education of any products or services offered or views expressed, nor does the 
Department of Education control its accuracy, relevance, timeliness or completeness. 
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